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ON  THE  POISSON  EQUATION 


FOR  COUNTABLE  MARKOV  CHAINS: 

EXISTENCE  OF  SOLUTIONS  AND  PARAMETER  DEPENDENCE 
BY  PROBABILISTIC  METHODS 


by 

Armand  M.  Makowski1  and  Adam  Shwartz2 


ABSTRACT 

This  paper  considers  the  Poisson  equation  associated  with  time-homogeneous  Markov  chains  on 
a  countable  state  space.  The  discussion  emphasizes  probabilistic  arguments  and  focuses  on  three 
separate  issues,  namely  (i)  the  existence  and  uniqueness  of  solutions  to  the  Poisson  equation,  (ii) 
growth  estimates  and  bounds  on  these  solutions  and  (iii)  their  parametric  dependence.  Answers  to 
these  questions  are  obtained  under  a  variety  of  recurrence  conditions. 

Motivating  applications  can  be  found  in  the  theory  of  Markov  decision  processes  in  both  its 
adaptive  and  non-adaptive  formulations,  and  in  the  theory  of  Stochastic  Approximations.  The 
results  complement  available  results  from  Potential  Theory  for  Markov  chains,  and  are  therefore  of 
independent  interest. 

Keywords  :  Markov  Chains,  Poisson  Equation,  smoothness  of  solutions. 


1  Electrical  Engineering  Department  and  Institute  for  Systems  Research  University  of  Maryland,  College  Park, 
Maryland  20742,  U.S.A.  The  work  of  this  author  was  supported  partially  through  NSF  Grant  ECS-83-51836,  partially 
through  ONR  Grant  N00014-84-K-0614  and  partially  through  United  States — Israel  Binational  Science  Foundation 
Grant  BSF85-00306. 

o 

Electrical  Engineering  Department,  Technion — Israel  Institute  of  Technology,  Haifa  32000,  Israel.  The  work  of 
this  author  was  partially  through  NSF  Grant  ECS-83-51836  and  partially  through  United  States — Israel  Binational 
Science  Foundation  Grant  BSF85-00306. 


1 


1.  INTRODUCTION 


Let  P  =  ( pxy )  be  the  one-step  transition  matrix  for  a  time-homogeneous  Markov  chain  {Xt,  t  = 
0,1,...}  taking  values  in  some  countable  space  S.  This  paper  is  devoted  to  the  corresponding 
Poisson  equation  with  forcing  function  c  :  S  — »  1R,  namely 


h(x)  -I-  J  =  c(x)  +  £  pxyh(y),  xeS  (1.1) 

for  scalar  J  and  mapping  h  :  S  i R.  This  equation  arises  naturally  in  a  variety  of  problems 

associated  with  Markov  chains  as  the  following  examples  indicate. 

1.  As  shown  in  Section  3,  solving  the  Poisson  equation  provides  a  means  to  evaluate  the 
long-run  average  cost  J  associated  with  the  cost  function  c  [26]:  If  (1.1)  has  a  solution  (h,  J)  and 
some  mild  growth  conditions  are  satisfied,  then  Lemma  3.1  states  that 


J  =  limt 


1 

t  +  1 


£<**.) 

s= 0 


(1.2) 


where  //  is  the  initial  distribution  and  is  the  corresponding  expectation  operator. 

2.  In  recent  years,  there  has  been  widespread  interest  in  stochastic  approximation  algorithms 
as  a  means  to  solve  increasingly  complex  engineering  problems  [1,12].  As  a  result,  focus  has 
shifted  from  the  original  Robbins-Monro  algorithm  to  (projected)  stochastic  approximations  driven 
by  Markovian  “noise”  or  “state”  processes.  Properties  of  solutions  to  an  appropriate  Poisson 
equation  play  an  essential  role  when  establishing  the  a.s.  convergence  of  such  adaptive  algorithms 
[1,13,17,19,20,29]. 

3.  In  the  context  of  Markov  decision  processes  (MDPs),  the  need  for  adaptive  policies  can  arise 
in  response  to  both  modeling  uncertainties  and  computational  limitations  [30].  Several  adaptive 
policies  have  been  proposed  as  “implementations”  to  a  Markov  stationary  policy,  and  shown  to  yield 
the  same  cost  performance  [3,  13,  14,18,30].  Here  too,  the  analysis  requires  precise  information  on 
the  solution  to  the  Poisson  equation  associated  with  the  non-adaptive  policy  [30]. 

In  many  of  these  applications,  it  is  appropriate  to  view  the  forcing  function  c  and  the  transition 
matrix  P  as  parametrized,  say  by  some  parameter  9  (which  may  be  loosely  interpreted  as  a  control 
variable).  The  analysis  then  typically  exploits  smoothness  properties  (in  9)  of  the  solution  h 
together  with  various  growth  estimates  (in  x)  for  h.  In  addition,  estimates  on  the  moments  of 
{h(Xt),  t  =  0, 1, 2, ... }  are  required,  with  the  added  difficulty  that  the  resulting  process  {Xt,  t  = 
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0, 1, 2, . . .}  is  not  necessarily  Markovian  (say,  under  the  given  stochastic  approximation  scheme  or 
adaptive  policy). 

In  this  paper,  our  main  objective  is  to  develop  methods  for  addressing  the  concerns  above  in  a 
systematic  fashion.  We  emphasize  a  probabilistic  viewpoint,  whenever  possible,  and  focus  mostly 
on  the  following  three  sets  of  issues,  namely 

1.  Existence  and  uniqueness  of  solutions  to  the  Poisson  equation  (1.1); 

2.  Growth  estimates  and  bounds  on  these  solutions;  and 

3.  Conditions  for  smoothness  in  the  parameter  of  these  solutions  when  dealing  with  the 
parametric  case  as  would  arise  in  establishing  the  a.s.  convergence  of  stochastic  approxi¬ 
mations. 

Answers  to  these  questions  are  given  under  a  variety  of  recurrence  conditions.  As  we  try  to  keep 
the  exposition  relatively  self-contained,  we  have  included  some  standard  material  on  the  Poisson 
equation.  In  addition  to  its  tutorial  merit,  the  discussion  given  here  provides  a  unified  treatment 
to  many  of  the  issues  associated  with  the  Poisson  equation,  e.g.,  existence,  uniqueness  and  repre¬ 
sentation  of  solutions.  This  is  achieved  by  manipulating  a  single  martingale  naturally  induced  by 
the  Poisson  equation. 

Questions  of  existence  and  uniqueness  of  solutions  to  (1.1)  have  obvious  and  natural  points 
of  contact  with  the  Potential  Theory  for  Markov  chains  [11,23].  However,  it  is  unfortunate  that 
many  situations  of  interest  in  applications,  say  in  the  context  of  MDPs,  are  not  readily  covered  by 
classical  Potential  Theory.  Indeed,  the  classical  theory  treats  the  purely  transient  and  recurrent 
cases  separately,  and  provides  drastically  different  results  for  each  situation.  This  approach  is  thus 
of  limited  use  in  the  above-mentioned  situations,  where  the  recurrence  structure  of  the  Markov 
chain  is  typically  far  more  complex  in  that  it  combines  both  transient  and  recurrent  states.  Here, 
in  contrast  with  the  analytical  approach  of  classical  Potential  Theory,  emphasis  has  been  put  on 
giving  an  explicit  representation  of  the  solution  to  (1.1)  with  a  clear  probabilistic  interpretation. 

This  probabilistic  approach  allows  for  a  relatively  elementary  treatment  of  existence  and 
uniqueness,  under  a  rather  general  recurrence  structure.  Results  are  obtained  in  various  degrees 
of  completeness  for  both  finite  and  countably  infinite  state  spaces;  recurrence  structures  include 
multiple  positive  recurrent  classes,  and  transient  classes.  A  representation  for  h  is  derived  in  detail 
in  the  case  of  a  single  positive  recurrent  class  under  integrability  conditions  involving  the  forcing 
function  c.  The  derivation  uses  elementary  methods,  and  provides  intuition  into  more  general  sit¬ 
uations.  This  representation  is  also  shown  to  hold  in  the  multiple  class  countable  case,  and  readily 


3 


lends  itself  to  establishing  natural  bounds  on  the  growth  rate  of  h  (as  a  function  of  the  state),  and 
to  investigating  smoothness  properties  in  the  parametrized  problem. 

The  paper  is  organized  as  follows:  The  set-up  is  given  in  Section  2,  together  with  the  basic 
martingale  associated  with  (1.1).  Various  uniqueness  results  on  the  the  solution  (J,  h)  are  discussed 
in  Section  3.  We  give  two  decomposition  results  in  Section  4;  one  such  decomposition  is  based  on 
the  decomposition  of  the  state  space  S  into  its  recurrent  and  transient  classes,  while  the  other  is 
an  analog  of  the  standard  Green  decomposition  and  relies  on  an  expansion  of  the  forcing  function 
in  terms  of  more  “elementary”  forcing  functions.  To  set  the  stage  for  the  countably  infinite  case, 
we  briefly  recall  an  algebraic  treatment  of  the  finite-state  case  in  Section  5.  In  section  6  an  explicit 
representation  for  the  solution  is  developed  in  terms  of  some  recurrence  times,  under  a  single 
positive  recurrent  class  assumption.  An  example  is  developed  in  Section  7  to  illustrate  the  material 
of  the  previous  sections.  Bounds  and  extensions  to  unbounded  cost  and  multichain  structures  are 
given  in  Section  8.  Equipped  with  this  probabilistic  representation  of  solutions,  we  embark  on 
investigating  smoothness  properties  of  the  solutions  to  the  parametrized  problem.  Methods  for 
proving  continuity  and  Lipschitz  continuity  are  developed  in  Section  9  and  10,  respectively. 

To  close,  we  note  that  most  of  the  ideas  which  are  discussed  here  in  the  context  of  countable 
Markov  chains  have  extensions  to  fairly  general  state  spaces.  This  is  achieved  by  means  of  the 
so-called  splitting  technique  [21,  22]  which  in  essence  guarantees  the  existence  of  an  atom. 

2.  THE  POISSON  EQUATION  AND  ITS  ASSOCIATED  MARTINGALE 

First,  a  few  words  on  the  notation  used  throughout  the  paper:  The  set  of  all  real  numbers  is  denoted 
by  R  and  1[A]  stands  for  the  indicator  function  of  a  set  A.  Unless  otherwise  stated,  limt,  fimt  and 
limt  are  taken  with  t  going  to  infinity.  Moreover,  the  infimum  over  an  empty  set  is  taken  to  be  oo 
by  convention.  The  mapping  J  :  S  x  S  — >  R  is  defined  by  S(x,y)  =  1  if  x  =  y,  and  S(x,y)  =  0 
otherwise.  Finally,  the  notation  J2x<=s  °ften  abbreviated  as  Y,x- 

2.1.  The  set-up 

The  notion  of  a  Markov  chain  we  adopt  here  is  more  general  than  the  elementary  one  used  in  most 
applications.  We  do  so  with  the  view  of  broadening  the  applicability  of  the  material  developed 
here,  especially  to  problems  of  adaptive  control  for  Markov  chains  [13,  14,  17,  18,  29,  30]. 

As  stated  earlier  in  the  introduction,  the  state  space  is  a  countable  set  S,  and  the  one-step 
transition  mechanism  is  given  by  the  5x5  stochastic  matrix  P  =  (pxy),  so  that  0  <  pxy  <  1  and 
EyPXy  =  1  f°r  ah  x  and  y  in  S.  We  assume  the  existence  of  a  measurable  space  (fi,  T)  large  enough 
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to  carry  all  the  probabilistic  elements  considered  in  this  paper.  In  particular,  let  [Xt,  t  =  0,1,...} 
denote  a  filtration  of  T ,  i.e.,  a  monotone  increasing  sequence  of  cr-fields  contained  in  J~  such  that 
Tt  C  Xt+i  for  all  t  =  0,1,...,  and  let  {Xt,  t  —  0, 1, . . .}  be  a  sequence  of  5-valued  rvs  which  are 
^-adapted,  i.e.,  the  rv  Xt  is  .Ty-measurable  for  all  t  =  0, 1, _ 

The  Markovian  structure  of  interest  is  defined  by  postulating  the  existence  of  a  family  (Px,  x  E 
5}  of  probability  measures  on  X  such  that  for  all  x  and  y  in  5,  we  have 

Pxpfo  =  y]  =  S(x,y)  (2.1  a) 

and 

P*[*t+1  =  y\Ti]=  PXty  P x-a.s.  t  =  0,1,...  (2.16) 

With  any  probability  distribution  y  on  5,  we  associate  a  probability  measure  PM  on  X  by  setting 

P„[A]  :='52XH(*)'P*[A],  (2.2) 

Obviously,  when  y  is  the  Dirac  measure  Sx  concentrated  at  some  x  in  5,  then  PM  reduces  to  Px. 
Using  (2. 1)— (2.2)  we  easily  see  that 

PM[X0  =  x]  =  y(x),  x  E  S  (2.3a) 

and 

P^[Xt+i  =  y  |  Tt\  =Pxty,  x,y  E  S  PM  -  a.s.  t  =  0, 1, . . .  (2.36) 

Under  Pfl  the  rvs  {Xt,  t  =  0, 1, . . .}  thus  have  the  Markov  property  with  respect  to  the  filtration 
{Xt,  t  =  0,1,...},  and  are  said  to  form  a  time-homogeneous  -Markov  chain  with  one-step 
transition  matrix  P  and  initial  probability  distribution  y.  In  many  instances,  we  take  Xt  to  be 
the  a-field  generated  by  the  rvs  Xo, . . . ,  Xt  for  all  t  =  0,1,...,  in  which  case  the  definition  above 
coincides  with  the  elementary  definition  of  a  Markov  chain. 

From  (2.1a)-(2.2)  we  readily  conclude  for  y- a.s.  all  x  in  5  that 

P^[ylj*o  =x]  =PX[A],  AEX  (2.4) 

so  that  Px  has  the  useful  interpretation  of  conditional  probability  (under  P;t  for  any  initial  distri¬ 
bution  measure  y). 

Throughout  it  will  be  convenient  to  denote  by  and  Ex  the  expectation  operator  associated 
with  PM  and  Px,  respectively. 
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2.2.  The  Poisson  equation 


Let  c  be  a  given  Borel  mapping  S  -¥  JR.  Throughout,  it  is  understood  that  a  constant  J  and  a 
mapping  h  :  S  — >  ]R  constitute  a  solution  pair  to  the  Poisson  equation  (1.1)  with  forcing  function 
c.  whenever  h  satisfies  the  integrability  conditions 

Y  pxv\h(y)\  <  oo,  xeS  (2.5a) 


and  the  relations 

h(x)  +  J  —  c(x)  +  Y  Pxyh(y),  x  e  S  (2.5 b) 

hold.  The  Poisson  equation  is  termed  homogeneous  if  c  =  0. 

For  any  initial  distribution  /i,  we  introduce  several  classes  of  IR-valued  mappings  defined  on 
S.  We  say  that  the  mapping  /  :  S  — >  1R  is  an  element  of 

1.  1^  if  E^l/pf*)!]  <  oo  for  all  t  =  0, 1, . . .; 

2.  B,j,  if  supt  EM[|/(Xt)|]  <  oo; 

3.  Sfj,  if  /  belongs  to  with  limt  jEM[/(Xt)]  =  0;  and 

4.  Ufj,  if  the  rvs  {f(Xt),  t  —  0, 1, . . .}  are  uniformly  integrable  under  PM. 

When  fj,  is  the  Dirac  measure  8X  for  some  x  in  S',  we  substitute  the  simpler  notation  Ix,  Bx,  Sx 
and  Ux  to  Xsx ,  Bf,x ,  Sgx  and  U$x,  respectively. 

For  any  initial  distribution  //,  we  readily  see  that 

U^CB^CS^C  (2.6) 

and  for  any  a;  in  S'  such  that  fi,{x)  >  0,  we  have  C  lx,  B^  C  Bx  and  Utl  C  Ux. 

Since  any  mapping  mapping  /  :  S  — >  1R  can  be  viewed  as  a  column  vector  (f(x  j),  the  Poisson 
equation  (1.1)  can  be  written  in  matrix  notation  as 

h  +  Je  =  c  +  Ph  (2.7) 

where  e  denotes  the  column  vector  with  all  its  entries  equal  to  one,  i.e.,  e(x)  —  1  for  all  x  in  S'.  For 
any  vector  /  =  ( f(x ))  and  any  subset  E  of  S,  denote  by  fs  the  restriction  of  /  to  E  and  similarly 
define  Pg  as  the  restriction  of  P  to  E.  The  identity  matrix  on  S  is  denoted  by  L 

2.3.  A  martingale  property 
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Many  of  the  general  results  on  solutions  to  the  Poisson  equation  can  be  traced  back  to  the  following 
observation. 

Lemma  2.1.  Let  the  pair  (h,  J )  be  a  solution  to  the  Poisson  equation  (2.5)  with  forcing  function  c. 
If  the  mapping  h  belongs  to  ZM  for  some  probability  measure  p  on  S,  then  the  following  statements 
hold: 

1.  The  forcing  function  c  is  necessarily  an  element  of  1^ ;  and 

2.  The  rvs  {Mt,  t  =  0, 1, . . .}  defined  by  Mo  h(X 0)  and 

t 

Mt+i  :=  h(Xt+ 1)  +  c(Xs )  —  (t  +  1)  J  t  =  0, 1, . . .  (2.8) 

s=0 

form  an  integrable  (P#t ,  Tt )  -martingale  sequence. 

Proof.  Invoking  the  Markov  property,  we  can  reformulate  the  Poisson  equation  (2.5)  as 

h{Xt)  +  J  =  c(Xt)  +  E/i[/i(Xt+i)|Z't]  t  =  0, 1, . . .  (2.9) 

and  the  P;i-integrability  of  the  rvs  {c(Xt),  t  =  0, 1, . . .}  follows  from  the  assumption  on  h.  This 
proves  Claim  1. 

To  establish  Claim  2,  we  first  conclude  from  the  first  part  of  the  proof  that  the  rvs  {Mt,  t  = 
0, 1, . . .}  are  well  defined  and  indeed  P;i -integrable.  From  (2.8),  we  then  get 

t 

EM[Mt+ i|JF t]  =  EM[/i(Xt+i)|Z"t]  +  c(Xs)  —  (t  +  1)  J  t  =  0, 1, . . .  (2.10) 

5  =  0 

because  the  rvs  X0, . . . ,  Xt  are  all  -measurable,  and  the  martingale  property  readily  follows  from 
(2.9).  ■ 


3.  UNIQUENESS  RESULTS 

In  this  section,  we  have  collected  several  uniqueness  results  for  the  Poisson  equation  (2.5).  In  that 
respect,  we  first  note  that  if  the  pair  ( h ,  J)  is  a  solution  to  the  Poisson  equation,  so  is  the  pair 
(h  +  ae,  J)  for  any  constant  a.  In  other  words,  uniqueness  can  only  be  obtained  up  to  an  additive 
constant.  We  also  observe  that  for  c  in  ZM,  the  definition 


J{p)  :=  limt  E^ 


1 

t  +  1 


s=0 


(3.1) 
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is  well  posed.  The  next  lemma  is  a  version  of  a  standard  result  from  the  theory  of  MDPs  under  a 
long-run  average  criterion  [9,26],  [30,  Lemma  3.1]. 

Lemma  3.1.  Let  the  pair  (h,  J)  he  a  solution  to  the  Poisson  equation  (2.5)  with  forcing  function 
c.  If  the  mapping  h  belongs  to  6),  for  some  probability  measure  /i  on  S,  then 


J  =  J(p)  =  lim* 


1 

t  +  1 


Ec<x.> 

s= 0 


(3.2) 


Proof.  Since  h  is  an  element  of  <SP,  it  is  also  an  element  of  1^  by  virtue  of  (2.6).  By  Claim  2  of 
Lemma  2.1  we  readily  obtain  the  equalities  E^[Mo]  =-  EM[Aft+i]  for  all  t  =  0, 1, . . .  or,  equivalently, 
in  expanded  form, 


Ep[h(X0)}^Ep(h(Xt+1)}+Ep 


£CPQ 

.4  =  0 


-(t+l)J. 


Some  simple  rearrangements  yield 


t  =  0,1,...  (3.3) 


E, 


=^-d-f{E„WX,+1)]~E„[A(A-o)]}  *  =  0,1,...  (3.4) 

4  =  0 


and  the  result  (3.2)  is  now  immediate  upon  letting  1 1  oo  in  (3.4)  since  h  is  an  element  of  .  ■ 

If  the  Poisson  equation  (2.5)  admits  a  solution  (h,J)  with  h  bounded,  then  c  is  bounded,  so 
that  both  c  and  h  belong  to  Utl  (thus  6’M)  for  any  initial  distribution  p.  It  then  follows  from  Lemma 
3.1  that  J(p.)  is  obtained  as  a  limit  which  does  not  depend  on  the  initial  distribution  p. 

The  uniqueness  of  solutions  to  the  Poisson  equation  is  now  briefly  studied  in  the  class  of 
“uniformly  Li -bounded”  solutions,  that  is  in  Bp  for  some  initial  state  distribution  p.  If  the  state 
space  contains  a  set  I  of  isolated  states  which  are  not  reachable  from  S\I  and  if  p(I)  =  0,  then 
clearly  the  chain  never  visits  the  states  in  J.  To  simplify  the  exposition  we  find  it  convenient  to 
reformulate  the  problem  on  the  reduced  state  space  S  —  I. 

The  next  lemma  is  preparatory  in  nature  and  will  greatly  simplify  the  presentation:  For  (hi,  Ji) 
and  (h2,  J2)  solution  pairs  to  the  Poisson  equation  (2.5),  we  define 


A  J  :=  Ji  —  J2  and  A  h(x)  :=  hi(  x)  —  h2(x),  x  €  S'. 


(3.5) 
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Lemma  3.2.  Let  (hi,  Ji)  and  (h2,  J2)  be  two  solutions  of  the  Poisson  equation  (2.5).  If  A h 
belongs  to  for  some  probability  measure  p  on  S,  then  the  rvs  {A  h(Xt)  —  A  Jt,  t  =  0, 1, . . .}  form 
a  (P^,  Pt) -martingale  sequence  with 

A J  =  ^{EM[Ah(Xt+,)]  -  EM[A/i(Xt)]}.  t  =  0, 1,  • . . ;  s  =  1, 2, . . .  (3.6) 


Proof.  Denoting  by  (Mtl,  t  =  0,1,...}  the  rvs  (2.11)  associated  with  the  solution  pair  (hi,  Jj), 
i  =  1, 2,  we  define  the  rvs  (A Mt,  t  =  0, 1, . . .}  by 

AM*  :=  Ml  -  Ml  =  Ah(Xt)  -  tAJ.  t  =  0, 1, . . .  (3.7) 

It  is  plain  that  (Ah,  A  J)  is  a  solution  to  the  homogeneous  Poisson  equation  Ah  +  AJe  =  P  Ah. 
Applying  Lemma  2.1  to  this  Poisson  equation,  we  conclude  that  the  rvs  {A Mt,  t  =  0,1,...} 
indeed  form  an  integrable  (P^X)) -martingale  sequence,  whence  EM[AMt+s]  =  E(1[AMt]  for  all 
s,t  =  0, 1, . . ..  In  expanded  form,  these  equalities  become 

E„[A/i(Xt+,)]  -  (t  +  s)AJ  =  EM[A/i(At)]  -tAJ  s,t  =  0, 1, . . .  (3.8) 

and  we  obtain  (3.6)  after  simple  rearrangements.  ■ 

The  basic  uniqueness  result  can  now  be  developed. 

Theorem  3.3.  Let  (hi,  J{)  and  (h2,  J2)  be  two  solutions  of  the  Poisson  equation  (2.5). 

1.  If  Ah  belongs  to  5/t  for  some  probability  measure  p  on  S,  then  Ji  =  J2; 

2.  If  in  addition  Ah  is  an  element  of  B^,  then  Ah  is  constant  on  each  recurrent  class  of  the 
Markov  chain  P(i. 

Proof.  If  Ah  belongs  to  <SM,  then  its  is  also  an  element  of  and  Claim  1  follows  by  letting  s  f  00 
in  (3.6)  and  using  the  fact  that  Ah  belongs  to  S^. 

The  proof  of  Claim  2  starts  with  the  observation  (2.6)  made  earlier  that  since  Ah  is  an 
element  of  B M,  it  it  is  also  an  element  of  S^.  Therefore,  J\  =  J2  by  Claim  1  and  the  rvs 
{A h(Xt),  t  =  0,1,...}  form  a  (P/1,Xt)-martingale  sequence  with  supt  EpIJAhpOl]  <  00.  By 
a  standard  martingale  convergence  theorem  [6,10],  the  martingale  sequence  {Ah(Xt),  t  =  0, 1, . . .} 
converges  PM-a.s.  to  a  proper  rv. 

If  all  the  states  in  S  form  a  single  recurrent  class  under  P,  then  any  two  states  in  S,  say  x 
and  y,  are  visited  infinitely  often  P^-a.s.  It  is  now  plain  that  h(x)  =  h(y)  by  virtue  of  the  P^-a.s. 
convergence  of  the  martingale  {Ah(Xt),  t  =  0, 1, . . .},  and  Ah  is  therefore  constant  on  S. 
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More  generally,  let  R  be  a  recurrence  class  under  P.  i.e.,  a  closed  irreducible  set  of  recurrent 
states.  Since  pxy  =  0  for  all  x  in  R  and  y  not  in  R,  (2.5)  implies 

hi,R  +  JeR  —  cr  +  PRhi'R,  i  —  1,2.  (3.9) 

The  matrix  Pr  can  be  interpreted  as  the  matrix  of  one-step  transition  probabilities  for  an  irre¬ 
ducible  Markov  chain  on  R  with  all  its  states  recurrent,  and  the  problem  is  now  reduced  to  the 
previously  considered  case.  Therefore,  h2tR  —  h2<R  is  constant  on  R  and  the  proof  is  complete.  ■ 

Under  conditions  weaker  than  the  ones  assumed  in  Theorem  3.3  we  can  obtain  a  refinement 
of  Claim  1  of  Theorem  3.3. 

Corollary  3.4.  Let  (hi,  Ji)  and  (h2,  J2)  be  two  solutions  to  the  Poisson  equation  (2.5).  If  for 
some  probability  measure  p  on  S,  hi  belongs  to  Sfl  and  h2  belongs  to  ,  then 

limt  -tEil[h2(Xt)]  =  J2-Ji  .  (3.10) 

Proof.  First  we  note  that  if  hi  is  an  element  of  SR  and  if  h2  belongs  to  then  Ah  belongs  to 
Xfl.  By  Lemma  3.2  we  get 

A J  =  i{E„[Afi(Xt)]  -  EM[Afi(A0)]}  *=1,2,...  (3.11) 

and  (3.10)  follows  upon  letting  1 1  °o  in  (3.11)  and  using  the  fact  that  hi  is  an  element  of  <SM.  The 
existence  of  the  limit  is  a  consequence  of  the  equalities  (3.11).  ■ 

It  is  very  easy  to  demonstrate  the  non-uniqueness  of  solutions  to  the  Poisson  equation:  Consider 
the  Markov  chain  P  =  (pXy)  on  the  non-negative  integers  IN  with  px,x+ 1  =  1,  x  =  0, 1, . . .,  and 
let  c  =  0.  Then  (h2,  J2)  =  (0,0)  is  obviously  a  solution  to  the  Poisson  equation  with  A 1  ( 0 )  =  0. 
However,  the  pair  (h2,  J2)  =  (x,  1)  is  also  a  solution  to  the  Poisson  equation  with  h2(0)  =  0.  For  all 
t  =  0, 1, . . .,  we  have  Xt  =  Xo+t  PM-a.s,  whence  ~Etl[h2(Xt)}  =  EM[X0]  +  f,  and  under  the  condition 
Em[X0]  <  00,  h2  is  an  element  of  2^,  but  not  of  Sfl.  In  fact,  (3.10)  holds  as  lim<  yE;t  [h2(Xt)}  = 
M0. 

In  Section  7  we  discuss  the  non-uniqueness  issue  for  a  more  elaborate  example  of  a  positive 
recurrent  system. 
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Although  in  practice  it  might  be  hard  to  verify  the  Li-boundedness  conditions  of  Theorem 
3.3,  a  simple  characterization  of  the  set  B^  is  available  in  a  special  yet  important  case.  Recall  that 
a  probability  measure  7  on  B(S)  is  an  invariant  measure  for  the  one-step  transition  matrix  P  if 


7(*)  =  'Y^,l{y)Pyx,  xeS.  (3.12) 

Under  P7  the  Markov  chain  { Xt ,  t  =  0,1,...}  forms  a  strictly  stationary  sequence  with  one¬ 
dimensional  marginal  distribution  7,  so  that  the  following  characterization  is  immediate. 

Lemma  3.5.  If  7  is  a 11  invariant  probability  measure  for  the  one-step  transition  matrix  P,  then 
J7  =  R7  =  =  Lx  (S,B(S), 7). 

In  [7]  Derman  and  Veinott  consider  the  uniqueness  issue  for  Markov  chains  with  a  single 
positive  recurrent  class  (in  which  case  the  invariant  measure  7  is  unique).  They  show  uniqueness 
in  the  class  DV  of  mappings  /  :  S  — >  1R  such  that 


E, 


T— 1 


E 1  /(*< 


t= 0 


<00,  x  e  S 


(3.13) 


where  T  inf{t  >  0  :  Xt  —  z}  for  some  distinguished  recurrent  state  2.  Under  these  assumptions, 
for  every  mapping  /  :  S  — >  1R,  we  conclude  by  standard  results  on  Markov  chains  [5]  that 

E7[|/(At)|]  =  •  ez[T] - *  *  =  0,1,...  (3.14) 

Therefore,  the  inclusion  DV  C  Bn  holds,  so  that  even  in  the  case  of  a  chain  with  a  single  positive 
recurrent  class,  the  conditions  of  [7]  are  more  stringent  than  the  ones  given  in  Theorem  3.3. 


4.  DECOMPOSITION  RESULTS 


4.1.  A  state  decomposition  result 

With  the  uniqueness  result  of  Theorem  3.3  in  mind,  we  consider  the  decomposition  of  the  countable 
set  S  induced  by  the  recurrence  structure  of  P:  Let  Tr  denote  the  (possibly  empty)  set  of  transient 
states,  and  let  {iia,  a  G  A},  for  some  countable  index  set  A,  denote  the  recurrent  components. 
The  sets  {Tr,  Ra,  a  G  A}  form  a  partition  of  S.  Moreover,  for  all  a  in  A,  pxy  =  0  for  x  in  Ra  and 
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y  not  in  Ra,  and  the  restriction  Pa  of  P  to  the  recurrent  class  Ra  is  irreducible  and  recurrent  on 
it.  With  the  vector  notation  of  Section  2,  (2.5)  can  now  be  partitioned  as 

hna  +  JeRa  =  cRa  +  PahRa,  a  e  A  (4.1a) 

hrr  +  JeTr  —  CTr  +  ^  , TahRa  +  Pt^Tt  (4-15) 

where  the  matrices  {Ta,  a  €  A}  and  Ptt  are  determined  from  the  decomposition  of  P  associated 
with  the  sets  { Tr,Ra ,  a  6  A}. 

The  decomposition  (4.1)  motivates  introducing  the  following  family  of  Poisson  equations 

ha  T  Joi&Ra  —  CRa  T  Paha,  cl  €  A  (4.2a) 

h  +  Jerr  =  c  +  Prrh  (4.26) 

where  for  each  a  in  A,  ha  is  a  mapping  Ra  1R,  while  c  and  h  are  mappings  Tr  — >  ]R,  with 

C  =  CTr  4"  ^  ^  ,  .  (4-3) 

The  next  result  shows  in  what  sense  the  solutions  to  the  projected  Poisson  equations  (4.2) 
determine  the  solution  to  the  original  equation  (2.5).  The  proof  is  a  simple  consequence  of  (4.1) 
and  (4.2),  and  is  omitted  in  the  interest  of  brevity. 

Theorem  4.1.  The  Poisson  equation  (2.5)  has  a  solution  if  and  only  if  the  following  two  conditions 
hold: 

1.  For  each  a  in  A,  the  Poisson  equation  (4.2a)  on  Ra  has  a  solution  (ha,Ja)  such  that 
Ja  —  J  for  some  scalar  J  independent  of  a  and 

y  '  .  Ta\ha |  <  oo;  (4.4) 

2.  The  Poisson  equation  (4.2b),  with  forcing  function  c  given  by  (4.3)  has  a  solution  (h,  J ) 
such  that  J  =  J . 

A  solution  pair  to  (2.5)  is  necessarily  of  the  form  ( h ,  J )  with  h  determined  by  hRa  =  ha  for  all  a 
in  A  and  hxr  =  h. 

Condition  (4.4),  which  is  automatically  satisfied  when  S  is  finite,  guarantees  that  c  (and 
therefore  (4.2b))  is  well  defined. 

4.2.  A  Green- like  decomposition 
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Let  (hi,  Ji)  and  (h2,  J2)  be  two  solutions  of  the  Poisson  equation  (4.2)  with  forcing  functions  c\ 
and  c2,  respectively.  Then  for  any  (3  in  R,  (h,  J )  :=  [phi  +  h2,  PJ\  +  J2)  is  a  solution  to  the  Poisson 
equation  (2.5)  with  forcing  function  fici  +  c2.  Indeed,  by  definition, 

h(x)  +  ./  =  (3(h\(x)  4-  Ji)  4-  (h2(x)  +  J2 ) 

=  P  (ci(s)  +  ^2yPxyhi(y)j  +  (c2(x)  +  ^2^pxyh2(y)^j 

=  (Pci(x)  +  C2(x ))  +  Y]  Pxy(Ph1(y)  +  h2(y)),  x  €  S  (4.5) 

y 

where  the  last  sum  is  well  defined  owing  to  the  definition  of  a  solution  (2.5b). 

This  simple  fact  can  be  used  as  follows:  For  each  v  in  S,  define  the  function  :  S  — 4  1R  by 
cv(x )  :=  S(v;  x)  for  all  x  in  S,  and  let  (hv,  Jv)  denote  a  solution  to  the  Poisson  equation  with  forcing 
function  cv.  The  obvious  decomposition 

c(x )  =  c(v)cv  (x)i  x  G  S  (4.6) 

then  leads  naturally  to  the  formal  representation 

J  =  Y1  c(v)Jv  and  h(x)  =  'Y2  c(v)hv(x),  x  E  S.  (4.7) 

It  remains  then  to  check  that  (4.7)  indeed  defines  a  legitimate  solution.  In  view  of  (4.5),  this  is 
the  case  whenever  c  is  constant  except  at  a  finite  number  of  points.  In  the  more  general  case,  this 
check  can  be  done  through  the  constructive  arguments  of  Corollary  6.2,  or  through  the  verification 
result  of  Theorem  6.4.  Such  a  calculation  is  performed  directly  in  Section  7. 

5.  FINITE  STATE  SPACES 

A  complete  picture  of  the  solution  to  the  Poisson  equation  (2.5)  is  available  when  S'  is  a  finite  set, 
and  can  be  found  in  [2,  31].  In  the  finite  space  case  any  solution  necessarily  belongs  to  for  every 
initial  probability  distribution  p.  Let  P*  denote  the  stochastic  matrix  defined  by 

p':=lim*  <5-') 

s=0 

its  existence  is  guaranteed  by  classical  results  from  the  theory  of  Markov  chains  [2,  31].  Because 
the  matrix  I  —  P  +  P*  is  invertible,  the  definition 

h:=(I-P  +  P*)~l(I-P*)c  (5.2) 
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is  well  posed.  The  easy  identities  P*P  =  PP*  =  P*P*  =  p*  lead  after  some  simple  algebra  to  the 
relation 

h  +  P*c  —  c  +  Ph.  (5.3) 

A  simple  comparison  of  (5.3)  with  (2.5)  suggests  that  h  defined  by  (5.2)  will  solve  the  Poisson 
equation  (2.5)  whenever  the  vector  P*c  is  proportional  to  e,  i.e.,  all  the  components  of  the  vector 
P*c  are  identical. 

To  investigate  the  matter  further,  we  introduce  the  canonical  decomposition  of  S  into  the 
recurrent  and  transient  components  induced  by  P,  as  already  done  in  Section  4.  Here,  it  can  be 
assumed  that  P  induces  m  recurrent  classes,  say  Pi, ... ,  Pm,  as  well  as  a  (possibly  empty)  set  Tr 
of  transient  states,  with  the  sets  {Pl5 . . .  ,  P,n,Tr}  forming  a  partition  of  S.  For  any  vector  /,  let 
fk  denote  the  restriction  of  /  to  Rk,  k  —  1, . . .  ,m. 

Recall  that  pxy  =  0  for  x  in  Rk  and  y  not  in  Rk,  and  the  restriction  Pk  of  P  to  the  recurrent 
class  Pfc  is  irreducible  and  positive  recurrent  on  it.  Possibly  upon  rearranging  P  into  a  block  lower 
triangular  form,  we  see  that  the  restriction  (P*)k  of  P*  to  Pfc  coincides  with  (Pfc)*  given  by 

1  , t 

(Pfc)*  limt  j-j-y  J^Pfc,  k  =  l,...,m  (5.4) 

5  =  0 

with  all  its  rows  being  identical  to  the  long-run  probability  distribution  associated  with  the  ir¬ 
reducible  chain  Pfc.  Consequently,  (P*c)fc  =  Jfcefc  where  the  scalar  Jk  depends  on  the  class  Pfc. 
Therefore  (5.3)  can  be  decomposed  as 

hk  +  Jkek  —  Ck  +  Pfc  hfc,  k  =  l,...,m  (5.5  a) 

m 

h-Tr  +  ( P*c)tt  =  Cjv  +  ^  ]  Tfc/lfc  +  Trh.Tr  (5.5 b) 

k— 1 

where  the  matrices  Ti, . . .  ,Tm  and  Tr  are  chosen  appropriately  from  the  decomposition  of  P  asso¬ 
ciated  with  the  sets  {Pi, . . . ,  Pm,  Tr}. 

Theorem  5.1.  The  pair  ( h,J )  is  a  solution  to  the  Poisson  equation  (2.5)  if  and  only  if  the 
conditions 

Ji  =  ...  —  ,Jm  =  J  and  (P*c)xr  =  Jexr  (5.6) 

hold,  in  which  case  h  is  given  uniquely  up  to  an  additive  constant  by  (5.2),  and  J  is  the  constant 
appearing  in  (5.6).  The  conditions  (5.6)  are  satisfied  when  the  Markov  chain  P  has  a  single  recurrent 
class. 
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Proof.  The  first  part  is  immediate  from  the  discussion  given  earlier  since  P*c  =  Je  under  (5.6). 
The  uniqueness  follows  from  Theorem  3.2  and  from  the  fact  that  I  —  Tr  is  invertible.  To  conclude 
the  last  part,  it  suffices  to  observe  that  under  the  assumption  of  a  single  recurrent  class  Ri  for  the 
Markov  chain  P,  the  rows  of  P *  are  all  identical  and  of  the  form  (j',  0 Tr)  where  v  coincides  with 
the  long-run  probability  distribution  vector  associated  with  the  irreducible  chain  Pi .  ■ 

In  fact,  (5.6a)  implies  (5.6b)  as  can  be  seen  from  the  discussion  in  Theorem  6.3  and  the  remark 
preceeding  its  proof. 

6.  A  PROBABILISTIC  FORMULA  FOR  SOLUTIONS 

Consider  now  the  situation  where  the  state  space  S  is  countably  infinite.  The  matrix  P*  is  still 
well  defined,  but  in  general  the  invertibility  of  I  —  P  +  P*  cannot  be  guaranteed  anymore  owing 
to  the  intricate  nature  of  the  recurrence  structures  for  Markov  chains  over  countably  infinite  state 
spaces.  As  a  result,  the  algebraic  discussion  of  Section  5  cannot  be  carried  through. 

In  some  situations  however,  probabilistic  arguments  can  be  used  to  prove  the  existence  of  a 
solution  pair  to  the  Poisson  equation.  Such  a  situation  arises  when  there  exists  a  distinguished 
state  in  5,  say  z,  which  is  positive  recurrent  in  a  sense  made  precise  below.  In  this  more  restricted 
set-up,  a  possible  approach  would  mimic  the  arguments  of  [26,  Section  6.7],  and  would  yield  the 
solution  as  the  limit  of  the  discounted  cost  associated  with  c,  when  the  discount  factor  tends  to  1. 
This  line  of  arguments  was  developed  in  [28]  and  does  yield  a  probabilistic  representation  of  the 
solution  already  obtained  by  Derman  and  Veinott  [7]  through  algebraic  means. 

In  this  paper,  we  take  a  different  route  for  deriving  this  probabilistic  representation  of  solutions 
to  the  Poisson  equation.  We  do  so  in  several  steps  by  exploiting  the  martingale  property  of  Lemma 
2.1.  To  precisely  state  the  conditions,  we  define  the  first  passage  time  to  the  state  z  as  the  Pt~ 
st  opping  time  T  given  by 

T  :=  inf{£  >  0  :  Xt  =  z}.  (6.1) 

The  recurrence  condition  (R)  enforced  thereafter  is  the  finite  mean  condition 

(R)  T(x)  :  =  Ex  [T]  <  oo,  x  6  S. 

The  condition  (R)  is  automatically  satisfied  when  the  set  S  is  finite  and  the  Markov  chain  P  admits 
a  single  (positive)  recurrent  class  decomposition  S  =  R  U  Tr  into  a  set  R  of  positive  recurrent 
states  and  a  (possibly  empty)  set  Tr  of  transient  states.  However,  when  the  set  S  is  not  finite,  the 
condition  (R)  is  far  more  stringent.  Indeed,  not  only  does  it  imply  the  single  class  decomposition 
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S  =  R  U  Tr,  but  it  also  prohibits  the  chain  from  wandering  too  long  or  exclusively  amongst  the 
transient  states.  We  relax  the  first  restriction  in  Section  8. 


We  also  find  it  convenient  to  consider  the  following  integrability  condition  (I),  where 


(I) 


C*(x)  :=  Ex 


T- 1 


E  icwi 


t= 0 


<  oo,  rr  6  S'. 


(6.2) 


Under  (I)  the  quantities 


rT— 1 


C(x)  :=  Ea 


E  CW 


Lt=o 


,  xe  s 


(6.3) 


are  well  defined.  Under  the  recurrence  condition  (R),  any  bounded  mapping  c  will  satisfy  the 
integrability  condition  (I);  in  fact  the  conditions  (R)  and  (I)  coincide  for  c(x )  =  1  for  all  x  in  S. 
The  next  result  is  a  consequence  of  the  martingale  property  given  in  Lemma  2.1. 


Theorem  6.1.  Assume  the  recurrence  condition  (R)  to  hold  and  let  (h,  J)  be  a  solution  pair  to 
the  Poisson  equation  (2.5).  If  h  is  an  element  ofXx  for  some  x  in  S,  then 


limn 


Ex  [1  [n  <  T]h(Xn)\  +  Ea 


T  An  — 1 


E  c (*o 


t- 0 


=  JT(x)  +  h(x)  —  h(z). 


(6.4) 


Proof.  By  Lemma  2.1,  the  rvs  {Mt,  <  =  0,1,...}  given  by  (2.8)  form  a  (Px, ^t)-martingale.  By 
Doob’s  Optional  Sampling  Theorem  [6,  10],  the  stopped  process  {MTAn,  n  —  0,1,...}  is  also  a 
(PxiRtau) -martingale,  so  that 


Ex[Mtau]  =  E  X[M0\  —  h(x). 


n  =  0,1,...  (6.5) 


By  Lemma  2.1  we  see  that  c  is  an  element  of  Xx  because  h  belongs  to  Xx,  and  therefore,  for  all 
n  =  0, 1, . . .,  the  three  rvs  h(XTAn),  T  An  and  Yl't=o~1  c(Xt)  are  integrable  under  Px.  From  the 
definition  of  Mr  An  we  conclude  by  direct  inspection  of  (6.5)  that 


h(x)  =Ea 


T  An— 1 

h(XTAn)  —  {T  An)J  +  E  c^*) 

t= o 


=h(z)Px[T  <  n]  +  Ex  [1[T  >  n\h(Xn)j  -  JEX[T  A  n]  +  Ex 


’T  An— 1 

E  c(x*) 

.  t= 0 


(6.6) 


Under  (R),  we  have  limn  Px [T  <  n]  =  1,  whereas  limn  Ex[TAn]  =  T(x)  by  monotone  convergence, 
and  the  result  (6.4)  follows  upon  letting  n  t  oo  in  (6.6).  ■ 
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As  we  impose  additional  conditions,  we  see  the  form  of  the  probabilistic  representation  emerge 
from  the  relation  (6.4). 

Corollary  6.2.  Assume  the  recurrence  condition  (R)  to  hold,  and  let  ( h ,  J)  be  a  solution  to  the 
Poisson  equation  (2.5).  If  h  belongs  to  Ux  for  some  x  in  S,  then  the  relation 


h(x)  =  limn  Ex 


~T  An— 1 

E  c(*<) 

.  t= 0 


-  JT(x)  +  h(z) 


holds.  If  in  addition,  the  integrability  condition  (I)  holds,  then 


(6.7) 


h(x)  =  C(x )  —  T(x)J  +  h(z). 


(6.8) 


Proof.  Under  (R),  we  have  limn  Pz[T  >  n]  =  0  and  the  uniform  integrability  of  the  rvs 
{h(Xt),  t  =  0,1,...}  under  Px  then  implies  lim„  Ex  [1[T  >  n]h(An)]  =  0,  so  that  (6.7)  follows 
from  (6.4).  Under  (I)  we  get 


limn  E* 


TAn-l 


E  c (**) 


t=0 


=  E« 


T— 1 


Ec(*<) 


t= 0 


=  C(x) 


(6.9) 


by  dominated  convergence,  and  (6.8)  is  an  immediate  consequence  of  (6.7)  and  (6.9). 


By  carefully  inspecting  this  last  proof,  we  can  extract  additional  information  on  the  interaction 
between  the  uniform  integrability  of  solutions  and  the  integrability  condition  (6.2):  We  define  the 
positive  and  negative  parts  of  the  forcing  function  c  by  c±(x)  :=  max{0,  ±c{x) }  for  all  x  in  S,  so 
that  c(x)  —  c+(x)  —  c-(x)  and  |c(m) |  =  c+(x)  +  c_(x).  In  analogy  with  (6.3),  we  introduce  the 
quantities 


rT-l 


C±(x)  :=  Ea 


E  c±(Xt) 


Lt= 0 


X 


eS 


(6.10) 


which  are  both  well  defined,  although  possibly  infinite.  The  relation  C (x)  =  C+  ( x )  —  (7_  (x)  holds 
provided  at  least  one  of  the  quantities  C+{x)  and  6Y_  (x)  is  finite,  while  the  equality  Ck{x)  = 
C+(x)  +  C-  (x)  is  always  valid. 

Corollary  6.3.  Assume  the  recurrence  condition  (R)  to  hold,  and  let  ( h ,  J)  be  a  solution  to  the 
Poisson  equation  (2.5).  If  h  belongs  to  Ux  for  some  x  in  S,  then  the  relation 


h(x )  +  C_(x)  -  C+(x)  -  JT(x)  +  h(z) 


(6.11) 
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holds.  If  in  addition,  c  is  either  bounded  above  or  below,  then  C\(x)  is  finite  and  the  relation  (6.8) 
holds. 


Proof.  We  go  back  to  the  proof  of  Theorem  6.1,  and  observe  that  for  all  n  =  0, 1 . . .  the  noted 
integrability  of  the  rv  Ylt=o~lc(xt)  implies  that  of  the  rvs  c±(Xt).  Using  this  fact  we 

can  rewrite  (6.6)  as 


h(x)  +  Ex 


"T  An— 1 

E 

.  t= o 


=h(z)Px[T  <  n]  +  Ex  [1[T  >  n]h(Xn)j  -  JEX[T  A  n]  +  Ex 


"T  Art  — 1 

E  c+™ 

.  t= 0 


(6.12) 


and  the  proof  now  proceeds  as  before:  Under  (R),  we  have  lim„  PX[T  <  n]  =  1,  and  limn  EX[T  A 
n]  =  T(x)  by  monotone  convergence.  Moreover,  the  uniform  integrability  of  the  rvs  {h(Xt),  t  = 

c±(xt) 

by  monotone  convergence.  The  result  (6.11)  follows  from  these  facts  upon  letting  n  t  oo  in  (6.12). 

To  establish  the  second  statement,  we  note  that  c  being  either  bounded  above  or  below  implies 
that  at  least  one  of  the  quantities  C+(x)  and  C-(x)  is  finite,  whence  both  are  necessarily  finite  in 
view  of  the  relation  (6.11).  ■ 

Corollary  6.2  states  that  under  conditions  (R)  and  (I),  any  “uniformly  integrable”  solution 
( h ,  J )  of  the  Poisson  equation  is  necessarily  given  by  (6.8)  (up  to  an  additive  constant).  In  a  sense, 
we  can  view  (6.8)  as  the  “minimal”  solution  to  (2.5).  However,  as  we  next  show,  (6.8)  does  define 
a  solution  even  when  there  may  exist  no  uniformly  integrable  one. 

Theorem  6.4.  Assume  both  the  recurrence  condition  (R)  and  the  integrability  condition  (I)  to 
hold.  Then  the  pair  (h,  J)  given  by 


0, 1, . . .}  under  Px  implies  lim„  Ex  [1[T  >  n]h(Xn)]  =  0,  and  C±(x)  =  lim„  Ex 


C{z) 

T(z) 


and  h(x)  =  C(x)  -  J  •  T(x), 


x  e  S. 


(6.13) 


is  a  solution  to  the  Poisson  equation  (2.5)  with  h(z)  =  0. 

When  the  state  space  S  is  finite  and  the  chain  has  a  single  recurrent  class,  (6.13)  provides  a 
probabilistic  interpretation  for  the  solution  described  through  purely  algebraic  means  in  [2,  31]. 

Although  condition  (R)  may  seem  quite  restrictive,  it  is  in  some  sense  close  to  being  necessary. 
Indeed,  as  shown  by  Cavazos-Cadena  [4,  Cor.  2. 1-2.2,  p.  105],  if  the  Poisson  equation  admits  a 
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bounded  solution  for  every  cost  c  which  vanishes  at  infinity,  then  (i)  there  exists  a  single  recurrent 
class,  which  is  necessarily  positive  recurrent;  and  (ii)  a  condition  stronger  than  (R)  holds,  namely 
sup„T(a;)  <  oo. 

Proof.  The  algebraic  manipulations  below  are  validated  through  the  following  summability  con¬ 
ditions 

E  -/  PxvT(y)  <  00  and  E  .  Pxy\C{y)\  <00,  x  €  S.  (6.14) 

In  view  of  the  comment  following  (6.3),  we  only  need  to  establish  the  second  condition  in  (6.14)  as 
the  first  one  reduces  to  it  when  c  =  1.  By  the  Markov  property,  we  get 

C*(x)  =  |c(z)|  +  E  ,  PxyC*{y),  x  e  S  (6.15) 

and  the  second  summability  condition  in  (6.14)  follows  from  the  integrability  condition  (I)  since 
\C(x)\  <  C+(x)  for  all  x  in  S. 

The  arguments  that  lead  to  (6.15)  also  show  that 

C{x)  =  c(x)  +  E  yitzPxyc(y)'  xeS  (6-16) 

and 

T(x)  =  l  +  Etf9t,P*»r(y),  xeS.  (6.17) 

For  any  scalar  J,  we  use  (6-16)— (6.17)  to  write 

C{x)  -  J  ■  T{x)  =  c{x )  +  E  y^PxyC(y)\  ~  J '  1  +  >  xeS  .  (6.18) 

Now,  with  the  choice  J  =  C(z)  jT{z),  (6.18)  becomes 

[C(x)  -  J  •  T(x)}  +  J  =  c(x)  +  Ev#,  Pxy[C(y)  -  J  ■  T(y)\ 

=  c(x)  +  E  p*y  _  J '  r(y)]>  x  e  s  (6-19) 

and  ( h ,  J)  is  indeed  a  solution  of  the  Poisson  equation  with  J  =  C(z)/T(z)  and  h(z)  =0.  ■ 

We  conclude  this  section  by  showing  in  what  sense  uniform  integrabilty  comes  close  to  being 
necessary  to  ensure  uniqueness.  This  will  follow  from  the  next  result  which  is  a  simple  consequence 
of  (6.4)  once  it  is  observed  that  C(x)  =  lim„Ex  |E^0n-1  c(Xt)l  whenever  C*(x)  is  finite. 
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Corollary  6.5.  Assume  the  recurrence  condition  (R)  to  hold  and  let  ( h ,  J )  be  a  solution  pair  to 
the  Poisson  equation  (2.5).  If  h  is  an  element  of  Xx  for  some  x  in  S  and  if  C*(x)  is  finite,  then 

lin^E*  [l[n  <  T}h(Xn )]  =  h(x)  -  h(z )  -  (C(x)  -  JT(x)\ .  (6.20) 

We  see  from  (6.20)  that  this  solution  h  in  lx  coincides  with  that  given  by  (6.13)  provided 
lininE*  [l[n  <  T]h(Xn)\  =  0,  a  condition  reminiscent  of  uniform  integrability  (i.e.,  h  in  Ux)  and 
indeed  implied  by  it. 

7.  AN  EXAMPLE 

In  this  section  we  specialize  the  results  obtained  so  far  to  a  simple  reflected  random  walk.  The 
solution  given  by  the  probabilistic  representation  is  computed  explicitly,  and  shown  to  belong  to 
B1  (—  U~/  where  7  is  the  invariant  distribution)  whenever  the  forcing  function  c  is  an  element  of 
B 7.  In  that  case,  we  also  iidentify  a  class  of  solutions  which  are  not  uniformly  integrable;  in  fact, 
we  calculate  all  solutions  to  the  Poisson  equation,  thereby  exhibiting  non-uniqueness  for  a  positive 
recurrent  Markov  chain.  The  calculations  are  carried  out  in  Appendix  A  below. 

The  situation  considered  here  is  that  of  a  random  walk  on  the  non-negative  integers  with 
reflection,  i.e.,  5  =  IN  and 

Po,o  =  Px+i,x  =  1  -  P  :=  q  and  Px,x+i  =  P  x  =  0, 1, . . .  (7.1) 

for  some  0  <  p  <  1.  With  a  queueing-theoretic  interpretation  in  mind,  we  define  p  :=  p/q,  and 
note  that  this  Markov  chain  is  positive  recurrent — and  condition  (R)  holds — whenever  p  <  1  (or 
equivalently  0  <  p  <  1/2).  In  that  case,  making  use  of  the  defining  relation  (3.16),  we  readily 
determine  the  invariant  distribution  7  to  be 

7(*)  =  (1  ~P)PX  •  x  =  0,l,...(7.2) 

For  any  forcing  function  c,  the  Poisson  equation  (2.5)  here  takes  the  form 

ph(0)  +  J  —  ph(  1)  +  c(0) 

and 

h(x  +  1)  +  J  =  qh(x)  +  ph(x  +  2)  +  c(x  +  1) .  x  =  0, 1, ...  (7.3) 

Before  addressing  the  existence  of  solutions  to  (7.3),  we  show  that  such  solutions  are  not  unique. 
Indeed,  if  (/ij,  Jj),  i  =  1,2,  are  two  solution  pairs  to  (7.3),  then  their  difference  (Ah,  AJ)  (in  the 
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notation  (3.5))  solves  the  homogeneous  equation  Ah  +  AJe  —  PAh,  which  can  be  rewritten  as 

p[Ah(l)  -  Ah(0)\  =  AJ 

and 


p  [A h(x  +  2)  —  A h(x  +  1)]  =  AJ  +  q  [Ah(o:  +  1)  —  A h(x)\ .  x  =  0, 1, . . .  (7.4) 


For  any  value  of  AJ  it  is  a  simple  matter  to  show  that  all  the  solutions  to  (7.4)  are  given  by 


A  h(x)  =  Ah(0)  + 


AJ 

p-q 


1  ~P~a 

L  1~p 


+  X 


,  x  6  S 


(7.5) 


and  parametrized  by  the  initial  condition  Ah(0).  Therefore,  if  (hi,  Ji)  is  a  solution  to  (7.3),  so  is 
(hi  +  Ah,  Ji  4-  AJ)  for  any  choice  of  AJ  (in  H)  where  Ah  is  given  by  (7.5)  with  that  value  of  AJ. 
In  other  words,  even  when  all  solutions  to  (7.3)  are  required  to  have  identical  initial  conditions — a 
normalizing  condition  which  dictates  Ah(0)  =  0  in  (7.5) — we  conclude  that  the  solution  set  to  (7.3) 
must  necessarily  be  non-countable  provided  it  is  not  empty.  This  non-uniqueness  is  independent  of 
the  choice  of  c,  and  holds  also  when  p  >  1,  i.e.,  the  chain  is  null  recurrent  or  transient. 


When  0  <  p  <  1,  we  observe  that  Ah  given  by  (7.5)  can  never  belong  to  U1  unless  AJ  =  0, 
thereby  confirming  the  uniqueness  of  solutions  in  h/7,  a  result  that  derives  from  Theorem  3.3  (and 
independently  from  Corollary  6.2).  Therefore,  it  now  remains  to  determine  conditions  under  which 
the  solution  in  U 7  exists. 

With  the  representation  (6.13)  in  mind,  we  take  z  =  0  and  use  (6.17)  to  obtain 


T(x) 


q8(  0,  x)  +x 
q-p 


®  =  0,1,...  (7.6) 


calculations  are  outlined  in  Appendix  A. 

Next,  intent  on  using  the  Green  decomposition  technique  of  Section  4.2,  we  compute  for  each 
v  in  S  the  cost  per  cycle  function  Cv  associated  with  the  cost  c„  :  S  — IR  :  x  — >  5  (v,x).  Since 
Jv  =  7(u),  we  invoke  (3.14)  to  get 


C„(0)  =  J„T(0)  =7(v)t±-  =PV- 

1  -  P 


(7.7) 


Cv{  x)  =  v/q 


*  =  1,2,...  (7.8) 


In  Appendix  A  we  also  show  that 

v  =  0,1  : 

v  =  2:  C„(l)  -  p2/p, 

v  =  3, . . .  : 

v  =  3, . . .  : 


Cv(x)  =  1/q2 

x  =  2, 3, . . 

.  (7.9) 

Cv(x)  =  p'/p * 

x  =  1,2 

(7.10a) 

..  xAu— 1 

c„M  =  -  E 

V  n—Pl 

x  =  3, . . . 

(7.105) 

3=0 
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Substituting  (7.6)-(7.10)  into  (6.13),  we  obtain  the  solution  hv  to  the  Poisson  equation  with 
forcing  function  c.v  in  the  form 

hv(x)  —  Cv(x)  —  JvT(x)  =  Cv(x)  —  pvx/q,  x  E  S.  (7.11) 

Inspection  of  (7.7)-(7.10)  reveals  that  Cv(x)  is  bounded  in  x,  and  the  solution  hv  thus  grows  linearly 
is  x.  Therefore,  invoking  Lemma  3.5  (in  conjunction  with  (7.2)),  we  see  that  hv  is  an  element  of 
U1  and  is  therefore  the  unique  solution  in  that  class. 

Using  the  Green  decomposition  technique  of  Section  4.2,  we  can  identify  a  large  class  of  forcing 
functions  for  which  (7.3)  will  have  a  unique  solution  in  Uy\  details  of  the  derivation  are  available 
in  Appendix  A. 

Theorem  7.1.  Consider  the  random  walk  with  reflection  at  the  origin  defined  through  (7.1)  with 
0  <  p  <  1.  Let  c  be  a  forcing  function  S  — >■  1R  such  that  |c(x)|  <  K(  1  +  rx)  for  all  x  in  S,  for  some 
positive  constants  r  and  K.  If  rp  <  1,  then  the  decomposition  (4.7)  (where  ( hv,Jv )  is  given  by 
(7.11)  for  all  v  in  S)  provides  a  solution  ( h ,  J)  to  the  Poisson  equation  (2.5),  and  this  solution  is  in 
Uy. 

8.  BOUNDS  AND  EXTENSIONS 

In  this  section,  we  explore  already  some  of  the  advantages  afforded  by  the  probabilistic  represen¬ 
tation  (6.13).  We  use  it  to  develop  various  bounds  on  the  solution  to  the  Poisson  equation  and  to 
obtain  an  existence  result  for  unbounded  costs  under  a  multichain  structure. 

8.1.  Bounds 

The  following  growth  estimate  is  an  easy  consequence  of  the  probabilistic  representation  (6.13). 

Theorem  8.1.  Assume  the  recurrence  condition  (R)  to  hold.  If  c  is  bounded,  i.e.,  A  := 
supx  |c(x)|  <  oo,  then  the  solution  pair  ( h ,  J)  given  by  (6.13)  satisfies  the  growth  estimate 

\h{x)\  <  (A  +  J)T(x),  x  E  S.  (8.1) 

Theorem  8.1  does  not  hold  when  c  is  not  bounded.  However,  in  many  situations  of  interest, 
the  underlying  Markov  chain  is  “skip-free  to  the  left”  with  respect  to  z.  For  example,  in  discrete¬ 
time  queueing  systems  it  is  often  the  case  that  the  decrease  per  unit  time  in  the  total  number  of 
customers  is  bounded  above  by  the  maximal  number  of  available  servers,  say  K.  As  a  result,  with 
2  representing  the  empty  state,  we  obtain  the  relation  \Xt\  <  KT,  0  <  t  <  T,  where  |At|  denotes 
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the  total  number  of  customers  at  time  f,  and  T  is  here  the  time  until  the  system  empties.  With 
this  in  mind,  we  introduce  the  following  condition:  There  exists  a  positive  constant  K  such  that 

P  x[d(z,Xt)<KT,  0  <  f  <  T]  =  1,  xeS  (8.2) 

for  some  metric  d  on  S.  Under  such  a  condition,  the  representation  (6.13)  implies  the  following 
bound. 

Theorem  8.2.  Assume  both  the  recurrence  condition  (R)  and  the  integr&bility  condition  (I)  to 
hold.  If  the  Markov  chain  satisfies  (8.2),  and  if  c  exhibits  the  growth  condition 

|c(z)|  <  A  (1  +  d(z,x)5),  x€S  (8.3) 

for  positive  constants  A  and  S,  then  the  solution  h  given  by  (6.13)  satisfies  the  growth  estimate 

\h(x)\<B(T(x)+Bx[Ts+1]),  xeS  where  B  =  max{A  +  J,AKS}.  (8.4) 

In  other  words,  the  growth  rate  of  h  is  determined  by  the  growth  rate  of  moments  of  T.  In  particular, 
Theorem  8.2  shows  how  moments  of  recurrence  times  can  be  used  to  check  that  the  solution  (6.13) 
indeed  belongs  to  B M  or  U,L  for  some  p.  Such  information  is  of  interest  when  studying  the  a.s. 
convergence  of  stochastic  approximations  schemes  driven  by  Markov  chains  [1,20,17]. 

Proof.  Note  that  (8.4)  is  automatically  satisfied  for  x  —  z  since  then  h(z)  =  0.  Now,  fixing  x  z 
in  S,  we  observe  from  the  definition  of  C(x )  that 

T— 1 

|C(x)|<E,  X>(X,)| 

.  t=0 
"T— 1 

<AEX  5^(1  +  d(z,Xt)5) 

_t= o 

("T— 1 

T(x)  +  Ex  ^2(kt)S 

_t= 0 

=  A  (T(x)  +  K5 Ex  [T5+1] )  (8.5) 

where  the  second  and  the  third  inequalities  were  obtained  by  making  use  of  (8.3)  and  (8.2),  respec¬ 
tively.  The  form  of  (6.13)  now  yields  (8.4).  ■ 

8.2.  Multiple  classes 
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When  the  state  space  contains  several  positive  recurrent  classes,  it  is  convenient  to  use  a  decom¬ 
position  of  the  state  space  S  into  its  transient  and  recurrent  components  {Tr,Ra,  a  E  A},  and  to 
partition  the  Poisson  equation  accordingly.  The  treatment  is  similar  to  the  one  sketched  briefly  in 

[31]. 

With  the  decomposition  and  notation  of  Section  4,  the  results  of  the  previous  section  extend 
to  the  multiple  class  case.  For  every  a  in  A,  select  a  state  za  in  Ra  and  define  the  first  passage 
times  to  the  states  za,  a  in  A,  and  to  the  set  Z  {za,  a  E  A}  by 

Ta  inf{t  >  0  :  Xt  —  za },  at  A  (8.6) 

and 

T  :=  inf {t  >  0  :XtEZ}.  (8.7) 

Since  each  recurrent  class  is  closed  under  P,  at  most  one  of  the  rvs  { Ta ,  a  6  A}  is  finite  Px-a.s. 
for  each  x  in  S,  so  that 


T  =  E  Tal[Ta  <  oq]  on  [T  <  oo]  Px  -  a.s.  (8.8) 

*  a 

under  the  convention  0  •  oo  =  0.  For  future  use,  we  also  define 


Ta{x)  :=  Ex[Tal[Ta  <  oo]],  a  €  A,  x  €  S.  (8.9) 

The  appropriate  version  of  condition  (R)  for  the  multiple  class  case  is  the  finite  mean  condition 
(Rm)  T(x)  :=  EX[T]  <  oo,  x  E  S. 

Note  that  (Rm)  is  essentially  (R)  but  with  the  first  passage  time  T  defined  through  (8.8)  rather 
than  by  (6.1).  Under  (Rm),  it  is  plain  that  for  each  x  in  S,  T  <  oo  Px-a.s.  and  that  for  each 
a  in  A,  Ta  {x)  =  E  x[Ta]  <  oo  whenever  x  lies  in  Ra  with  the  implication  that  all  recurrent  states 
are  positive  recurrent.  Condition  (Rm)  also  implies  that  starting  at  any  state  x  in  5,  the  process 
eventually  reaches  the  recurrent  classes  and  does  so  in  finite  expected  time. 

We  now  impose  conditions  (Rm)  and  (I)  (with  T  defined  through  (8.8)).  For  every  a  in  A, 
the  following  expressions 


Ca(x)  =  Ex 


~Ta- 1 

E  c(**) 


L  t= o 


X  E  Ra 


and  Ja  := 


C<x 

Ta(za) 


are  then  well  defined. 


(8.10) 
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Theorem  8.3.  Assume  the  recurrence  condition  (Rm)  and  the  integrability  conditions  (I)  to 
hold.  If  there  exists  a  scalar  J  such  that  Ja  =  J  for  all  a  in  A,  then  the  pair  ( h ,  J)  with  h  :  S  -*  1R 
given  by 

h(x)  =  C(x)  —  J  ■  T(x),  x  in  S  (8.12) 

is  solution  to  the  Poisson  equation  with  the  property  that  h(z )  =  0  for  every  z  in  Z. 

Proof.  The  proof  proceeds  in  two  steps. 

Step  1:  First  assume  the  set  Tr  of  transient  states  to  be  empty.  In  that  case  the  result 
follows  readily  from  Theorem  6.3  if  it  can  be  shown  that  for  each  a  in  A,  the  pair  (hRa,  .Ja)  is 
indeed  a  solution  pair  to  the  projected  Poisson  equation  (4.5a)  on  Ra.  That  this  is  indeed  the  case 
can  be  seen  as  follows.  The  recurrence  condition  (Rm)  implies  that  the  restriction  of  the  Markov 
chain  P  to  the  recurrence  class  Ra  satisfies  the  condition  (R)  imposed  in  the  single  recurrent  case. 
Therefore,  by  Theorem  6.3  the  projected  Poisson  equation  (4.5a)  on  Ra  admits  as  solution  the  pair 
{ha,Ja)  given  by 

ha{x)  =  Ca( x)  -  Ja  ■  Ta(x),  x  E  Ra  (8.13) 

with  Ja  given  by  (8.11).  However,  under  (Rm)  note  that  for  x  in  Rn,  T  =  Ta  <  oo  Px-a.s., 
whence  T(x)  =  Ta(x)  and  C(x)  —  Ca(x).  As  a  result,  we  find  that  J  =  .Ja  =  ^  so 

that  indeed  h(x)  ha(x)  for  x  in  Ra. 


Step  2:  When  Tr  is  not  empty,  the  difficulty  in  obtaining  a  solution  to  the  Poisson  equation 
is  related  to  the  existence  of  transient  states  from  which  more  than  one  recurrent  class  can  be 
reached.  First  observe  however  that  now  (6.13)-(6.14)  have  to  be  replaced  by 


T{x)  =  1  +  d7PxyT{y),  xeS 

(8.15) 

and 

C(x)  =  c(x)  +  ^2yizPxyC(y),  XES. 

(8.16) 

Therefore,  in  the  same  way  that  (6.13)-(6.14)  lead  to  (6.16),  it  is  easy  to 

imply 

see  that  (8. 15)— (8. 16) 

[C{x)  -  J  ■  T(x)}  +  J  =  c(x)  +  Y,yizPXy[C(y)  -  JT(y )] 

=  c(x)  +  ^^[C(y)  -  J  ■  T(y)\, 

x  e  S  (8.17) 

where  the  last  step  follows  from  the  fact  that  C(z)  =  J  ■  T(z)  for  every  z  in  Z  as  was  noted  in  the 
first  part  of  the  proof.  This  time  algebraic  manipulations  are  validated  through  the  summability 
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conditions 


EytzPxyTM  < 00  and  ^yiZPxy\C{y)\<oo,  xeS  (8.18) 

which  follow  from  (8. 15)— (8. 1 6)  and  the  integrability  condition  (I).  ■ 

It  is  easy  to  see  that  in  this  case,  (3.10b)  also  has  a  solution,  using  (3.10a)  and  the  fact  that  for 
all  x  in  S  and  y  in  Ra,  the  n-step  transition  probabilities  pxy  each  converge  to  Px \Ta  <  00]  • 
where  1/“)  is  the  invariant  distribution  of  the  Markov  chain  P  when  restricted  to  Ra. 

9.  PARAMETRIC  DEPENDENCE:  CONTINUITY 

In  several  applications,  including  stochastic  adaptive  control  and  stochastic  approximations  [1,  3,  13, 
17,  19,  20],  the  analysis  simultaneously  deals  with  a  parameterized  family  of  Markov  chains,  rather 
than  a  single  Markov  chain,  and  crucial  to  the  arguments  is  the  smoothness  (in  the  parameter)  of 
solutions  to  the  associated  Poisson  equations.  Of  particular  interest  are  conditions  on  the  model 
data  which  guarantee  that  the  solution  to  the  Poisson  equation  is  continuous,  or  even  Lipschitz 
continuous  in  the  parameter.  It  is  the  purpose  of  this  and  the  next  sections  to  show  how  the 
representation  results  of  Sections  6  and  8  provide  a  natural  vehicle  to  explore  this  question.  Our 
intent  is  not  to  get  the  best  possible  results,  but  rather  to  suggest  ways  of  attacking  these  parametric 
issues. 

In  order  to  simplify  the  notation,  the  discussion  is  given  only  for  the  case  of  a  scalar  parameter 
set,  as  similar  arguments  can  be  developed  mutatis  mutandis  for  more  general  situations:  Let  the 
parameter  set  0  be  a  Borel  subset  of  1R,  and  consider  a  family  {P{6),  9  6  0}  of  one-step  transition 
probability  matrices  on  the  countable  set  S,  with  P(9)  =  {pxy{9))-  For  every  9  in  0  and  x  in  5, 
let  P®  and  denote  the  probability  measure  and  corresponding  expectation  operator  induced  on 
(£2,  P)  by  P(8)  given  that  X0  =  x. 

The  set-up  is  the  one  of  Section  6.  There  exists  a  distinguished  state  z  in  S  such  that  for  all 
9  in  0, 

T(9,  x )  :=  E ex[T]  <00,  x  <E  S  (9.1) 

with  T  still  denoting  the  first  passage  time  (6.1)  to  the  state  z.  In  other  words,  the  recurrence 
condition  (R)  holds  with  respect  to  the  same  point  z  independently  of  9. 

For  every  9  in  0,  a  given  mapping  c{9)  :  S  — >  IR  :  x  — >  c,(9,  x )  is  assumed  to  drive  the  Poisson 
equation  (2.5)  associated  with  P(6),  i.e., 

h  +  J  =  c{9)e  +  P{9)h.  (9.2) 
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For  all  0  in  0,  the  integrability  condition  (I) 


E? 


T— 1 


E  W-**)! 


t= 0 


<  oo,  x  E  S 


(9.3) 


is  assumed  to  hold.  Under  the  enforced  assumptions,  we  may  invoke  Theorem  6.3  to  conclude 
that  (9.2)  admits  at  least  one  solution  ( /i(0),  J(0 ))  where  J(0)  is  a  scalar  and  h(8)  is  a  mapping 
S  — »  1R  :  x  — >  h(8,x).  With  the  requirement  h(0,z)  =  0,  this  solution  ( h(8),,J(8 ))  has  the 
representation 

J(0)  =  and  h{8,  x )  =  <7(0,  x)  -  J(8)  ■  T(8,  x),  x  €  S.  (9.4) 

±{8,z) 


The  next  result  identifies  a  set  of  natural  conditions  for  establishing  continuity  of  solutions  to  (9.2). 
Such  a  regularity  property  was  required,  for  example  in  [3]. 

Theorem  9.1.  Under  the  foregoing  conditions,  suppose  that  for  each  x  in  S, 

(i)  the  mapping  8  — >  c(0,  x)  is  continuous  on  0; 

(ii)  the  mapping  0  ->  pxy(6)  is  continuous  over  ©  for  all  y  in  S; 

(iii)  the  family  of  probability  measures  {px.(0),  0  E  &}  on  S  is  tight; 

(iv)  the  rvs  {(T,  P®),  0  6  0}  are  uniformly  integrable;  and 

(v)  the  rvs  {(Y)J=o  |c(0,  Xt)  |,  Pf),  0  €  0}  are  uniformly  integrable. 

Then  for  every  x  in  S,  the  mappings  0  — »  T(0,  a r)  and  0  — >  <7(0,  x)  are  continuous  over  0. 

In  many  applications,  c(0,  x)  =  c(x)  for  all  x  in  S  and  0  in  0  so  that  (i)  automatically  holds, 
while  (iii)  is  satisfied  whenever  one-step  transitions  have  some  uniform  (in  0)  nearest-neighbor 
properties.  The  conditions  (iv)-(v)  are  usually  checked  by  (stochastically)  bounding  the  original 
system  uniformly  in  0  by  means  of  another  system  which  is  naturally  suggested  by  the  original 
system.  This  approach  was  taken  by  Rosberg  and  Makowski  in  [25]. 

The  next  two  lemmas  are  needed  in  the  proof  of  Theorem  9.1;  their  proof  can  be  found  in 
Appendix  B. 

Lemma  9.2.  Assume  (ii)-(iii)  of  Theorem  9.1.  For  all  x  and  y  in  S,  and  k  =  1, 2, . . .,  the  mappings 
0  — >  P^[T  —  k]  and  0  — >  F6x[Xt  —  y,T  =  k ],  1  <t  <  k,  are  all  continuous  on  0. 

Lemma  9.3.  Assume  (iii)  of  Theorem  9.1.  For  each  t  =  1,2,...  and  x  in  S,  the  family  of 
distributions  {(Xt,Px),  0  E  0}  is  tight. 


27 


To  prepare  the  proof  of  Theorem  9.1,  we  set 


Cm(0,x)  :=  E* 


TAm-l 

1  [T<m]  c&xt)  , 

t= o 


xes. 


m  =  1,2,...  (9.6) 


Proof  of  Theorem  9.1.  Let  £  be  a  fixed  element  in  S.  By  a  standard  decomposition  argument, 
there  is  no  loss  of  generality  in  assuming  c(8,  x)  >  0  for  all  a;  in  S'  and  8  in  0.  Moreover  the  first 
claim  follows  from  the  second  one  upon  using  c(8,x)  =  1. 

In  the  general  case,  standard  facts  from  analysis  [27]  imply  the  desired  continuity  result  if  it 
can  be  established  that  the  mappings  8  — V  Crn(8,x),  m  =  1,2, . . .,  are  continuous  on  0,  and  then 
that  the  convergence  limTOCm(0,  x)  =  C(8,x)  is  uniform  in  8. 

To  establish  the  first  step,  it  suffices  to  show  that  the  mappings  8  — »  Eex  [1[T  =  k]c(8,  Xt)], 
0  <  t  <  k,  are  continuous  for  (9.6)  can  be  written  as 


Cm(8, *)  =  £  £  E«  [1[T  =  k\c{0, Xt)} . 

fc=l t= o 


m  =  1,2,...  (9.7) 


Fix  0  <  t  <  k.  Because  the  rvs  {(Xt,P®),  0  6  0}  are  tight  by  Lemma  9.3,  for  every  <5  >  0  there 
exists  a  finite  subset  Gx(8)  of  S  such  that  sup9ee  P ex[Xt  &  Gx(^)]  <  8.  Therefore  the  easy  bound 


E*  [1[T  =  k] l[Xt  $  Gx(S))c(8,Xt)}  <  Eex 


T- 1 

1  [XtgGx(*)]£c(0,X,) 

s=0 


(9.8) 


and  the  uniform  integrability  condition  (v)  together  imply  that  for  every  e  >  0  there  exists  some 
5(e)  >0  such  that 

sup*  El  [1  [T  =  k]l(Xt  £  Gx(5(e))]c(8,Xt)}  <  e.  (9.9) 

On  the  other  hand,  the  mapping  8  — ►  E^  [1[T  =  k] l[Xt  6  Gx(5(£))]c(6,  Xt)]  is  continuous  by  virtue 
of  Lemma  9.2  since  Gx(5(e))  is  finite.  The  desired  continuity  of  the  mapping  8  — >  E®[1[T  = 
k\c(8,Xt)]  readily  follows  from  this  remark  and  from  (9.9)  by  using  a  standard  decomposition 
argument.  Details  are  left  to  the  interested  reader. 

For  the  second  step,  start  with  the  estimate 


0  <  C(8,x)  -Cm(8,x)  =  Eex 


T-l 

l[m<T]J2c(0,xt) 


t= o 


m  =  1,2,...  (9.10) 
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and  observe  that  the  uniform  integrability  of  the  rvs  {(T,P®),  9  E  0}  yields  lirrimSup^P^T  >  m]  = 
0.  This  fact  and  the  uniform  integrability  condition  (v)  immediately  imply  the  uniform  convergence 


lim™  sup,?  Ef  l[m  <  T]£C(0,X*)  =0 


(9.11) 


and  the  proof  is  now  complete.  ■ 

10.  PARAMETRIC  DEPENDENCE:  LIPSCHITZ  CONTINUITY 

Metivier  and  Priouret  [20]  have  shown  that  the  a.s.  convergence  of  stochastic  approximations  passes 
through  the  Lipschitz  continuity  of  solutions  (h(9),  J{9))  to  the  parametrized  Poisson  equation  (9.2). 
Arguments  for  establishing  such  Lipschitz  continuity  are  now  outlined  in  a  somewhat  restricted  set¬ 
up  which  often  occurs  in  applications  [13,  17].  To  that  end,  we  postulate  that  for  all  x  in  S ,  the 
probability  measures  {px-(9),  9  E  0}  on  S  are  mutually  absolutely  continuous,  i.e.,  if  pxy{9)  =  0 
for  some  y  in  S  and  9  in  0,  then  pxy{9')  =  0  for  all  9'  in  0.  As  a  result,  for  each  m  =  1,2,..., 
the  probability  measures  { P® ,  9  E  0}  are  mutually  absolutely  continuous  on  the  (7-field  Tm.  If 
^(9,9')  denotes  the  Radon-Nikodym  derivative  of  Pf’  with  respect  to  P®  (on  Tm),  then 


rx  (a  o')  TT  PXiXi+AQ') 


m  =  1,2,...  (10.1) 


where  the  convention  ^  =  0  is  adopted.  With  L%{9,9')  =  1,  the  rvs  {L^(0, 9'),  m  =  0, 1, . . .}  form 
a  (P^,  ^(-martingale,  and  for  any  non-negative  iPxAm-measurable  rv  X, 


K  [X}  =  Eex[L*TAm(0,9').X] 


m  =  0,1,...  (10.2) 


by  standard  results  on  absolutely  continuous  changes  of  measures  [6]. 

Theorem  10.1.  Under  the  foregoing  conditions,  suppose  there  exist  a  constant  K  >  0  and  a 
mapping  S  — »  (0,  oo)  :  x  —>  K(x)  such  that  for  all  9  and  9'  in  0, 

\pxy{9)-pXy{9')\<Kpxy(9)-\9-9'\,  x,y  E  S  (10.3) 


|c(0,  x)  —  c(9', x)\  <  K{x)  •  \9  —  9' \,  x  E  S. 


If  the  moment  conditions 


"T— 1 

K{x)  :=  sup#  Ef  ^  K(X '*)  <  oo.  *  G  S 

_t= 0 


(10.4) 


(10.5) 
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<  oo,  x  E  S 


(10.6) 


(l  +  *fX>(0,*t)| 

t= o 

are  satisfied  for  some  0  <  S  <  1,  then  for  every  x  in  S,  the  mappings  9  — >•  C(9,x)  are  locally 
Lipschitz  continuous  over  @.  In  fact,  whenever  \6  —  Q'\  <  ■£,  the  Lipschitz  estimates 

\c{9,x)-c(e',x)\<L(x)\9-e,\,  xes  (10.7) 

hold  with  L(x)  :=  KC(x)  +  K(x)  for  all  x  in  S. 

A  few  observations  are  in  order  before  giving  a  proof  of  Theorem  10.1:  A  result  on  the  Lipschitz 
continuity  of  the  mappings  9  — )■  T(9,x),  x  in  S,  is  readily  obtained  from  Theorem  10.1  upon  using 
c(9,x)  =  1,  in  which  case  conditions  (10.4)-(10.5)  are  automatically  satisfied,  and  (10.6)  reduces 
to 

f(x)  :=  sup0  T2  (1  +  5)rj  <  oo,  x  €  S.  (10.8) 

In  fact,  (10.6)  also  reduces  to  (10.8)  whenever  the  cost  function  is  bounded,  i.e.,  \c(9,x)\  <  B  for 
all  x  in  S  and  9  in  ©. 

When  the  Lipschitz  constant  in  (10.4)  does  not  depend  on  x,  i.e.,  K(x)  =  K  for  all  x  in  S, 
then  (10.5)  reduces  to  the  condition  sup0  [T]  <  oo  for  all  x  in  S. 

The  uniform  bounds  (10.8)  can  be  checked  in  a  variety  of  ways.  For  instance,  in  [15,  16,  17,  29] 
the  authors  considered  a  particular  model  where  the  distribution  of  the  first  passage  time  T  under 
P®  is  independent  of  9  -  of  course  a  rare  occurrence  -  so  that  (10.8)  becomes  a  simple  moment 
requirement.  Some  general  methods  are  sketched  in  [17].  In  other  situations,  specific  arguments 
have  to  be  developed,  as  we  now  do  under  the  assumption  that  for  some  distinguished  9*  in  ©, 
there  exists  a  constant  B  >  0  such  that  for  all  6  in  ©, 

<  B  whenever  pxy(9*)  >0,  x,y  6  S.  (10.9) 

In  that  case,  fixing  9  in  0  and  x  in  S,  we  observe  from  (10.2)  that 

E*  [1[T  <  m]  ( T  A  m)]  <  Ef  [l[T  <  m](T  A  m)  •  BTAm }  m  =  1,2,...  (10.10) 

because  0  <  Lj,Am(9*,9)  <  BTAm  by  virtue  of  (10.9),  whence  Ef  [T]  <  Ef*  [T  ■  BT]  by  a  sim¬ 
ple  limiting  argument.  The  same  reasoning  shows  that  E®  [T2(l  +  ^)T]  <  E^*  [T2((l  +  S)B)r] . 


C{x)  :=  suPe  E^ 
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Consequently  (10.8)  holds  under  the  structural  condition  (10.9)  if  the  more  compact  conditions 
Ef  [T2((  1  +  S)B)t]  <  oo  holds. 

Proof  of  Theorem  10.1.  Let  a;  be  a  fixed  element  in  S.  As  in  the  proof  of  Theorem  9.1,  there  is 
no  loss  of  generality  in  assuming  c(9,  x)  >  0  for  all  a;  in  S'  and  9  in  0. 

Fix  6  and  O'  in  0.  It  is  easily  seen  from  (9.6)  and  (10.2)  that 


Cm{0',x )  :=  E* 


T  Am  —  1 

l[T<m\.LxTAm(9,9')  E 

t= o 


m  =  1,2,. 


.(10.11) 


With  this  relation  in  mind,  we  define 

Am{9,9'):=V6x  l[T<m}[l-LxTATn(0,e')}-  E  c(^Xt) 


TAm-1 


(10.12) 


and 


SUM')  :=E® 


1  [T<m\-LxTAm(9,9') 


t= 0 


T  Am— 1  TAm-1 


E  WM-  e 


i= 0 


t=  0 


(10.13) 


for  all  rn  =  1,2,...,  so  that  Cm(9,x )  -  Cm(9',x)  =  Am{6,6')  +  Bm(9,6')  for  m  =  0,1,.... 
Condition  (10.3)  implies 

1  _  Pxy(Q') 

Pxy{9) 


<  K  •  \0  —  9' \  whenever  pxy{9 )  >0,  x,y  6  5, 
so  that  on  the  event  [Lj.Am(0, 9')  >  0],  provided  A|0  —  9'\  <  1, 

(1  -  K\9  -  0'|)TAm  -  1  <  LxTAm{9, 9')  -  1  <  (1  +  K\9  -  9'\fAm  -  1.  m  =  1, 2, . 

From  the  easy  identities 

{l±Kt)m-l=[  (BmK)  ■  (1  ±  Kr)m~ldT,  t>0  m=l,2,. 

Jo 

we  conclude 

1(1  ±  Kt)TAm  —  l|  <  K(T  A  m)  •  (1  +  5)TAm  ■  t  m=  1,2,. 


(10.14) 


(10.15) 


(10.16) 


(10.17) 


whenever  0  <  t  <  j?  (where  0  <  6  <  1).  Therefore,  upon  combining  (10.15)  and  (10.17),  under  the 
condition  K\9  —  9'\  <  8  we  find 


\Am(6,0')\<K--Eex 


T  Am—  1 


l[T<m]-(TAm)-(l  +  6)(TAm)-  E  c(°iXt) 


t= o 


\9-9'\ 
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for  all  m  —  1,2,...,  and  a  simple  limiting  argument  yields 


linw  \Am(9,9')\  <  A-Ef 


T- 1 


T-(1  +  S)t  •  J>(0,Xt) 


t=o 


\9  —  G'\. 


(10.18) 


On  the  other  hand,  we  have 
\Bm(9,9')\<E°x 


TAm—l 


<K 


l[T<m\.L*TAm(9,0')-  Y  |  c(0,Xt) 

t= 0 

TAm—l 

l[T<m}.L*TAm(0,0')'  Y,  R(X*) 


t= o 


=  e: 


O' 


TAm—l 


1  [T<m]  Y  K(X<) 


t= 0 


\e-e'\ 


-c{d',Xt)  | 

•  \e-9'\ 

m  =  1,2,...  (10.19) 


where  the  second  inequality  is  a  consequence  of  (10.4),  and  the  final  equality  follows  from  (10.2). 
Consequently,  in  the  limit,  we  conclude  that 


* 


lim™  |Bm(0,«')l  <  Ef 


Lt=o 


1 9-9' 


and  the  result  now  readily  follows  from  (10.18)  and  (10.20). 


(10.20) 


Acknowledgment:  We  are  indebted  to  an  anonymous  referee  for  pointing  out  reference  [7]. 


APPENDIX  A:  THE  EXAMPLE 

To  obtain  (7.6)  from  (7.2),  we  apply  (6.17)  with  z  =  0  to  get 


T(x)  =  l+pT(x  +  l), 

x  =  0, 1 

(A.1) 

T(x)  =  1  +  pT{x  +  1)  +  qT(x  -  1). 

x  =  2,3,... 

(A.2) 

Since  T(0)  =  1/7(0)  by  standard  results  on  Markov  chains,  we  can  use  (7.2)  to  obtain  (7.6). 
Indeed,  the  validity  of  (7.6)  can  be  seen  by  substituting  T(0)  into  (A.1)-(A.2),  so  that  T(3)  — 
T{ 2)  =  T( 2)  —  T(l).  Using  (A. 2),  we  see  that  this  last  equality  propagates  by  induction,  i.e., 
T(x  +  1)  —  T(x)  =  T(x)  —  T(x  —  1)  for  all  x  =  2, . . .  and  (7.6)  readily  follows. 


32 


Fixing  v  in  S,  we  now  set  out  to  compute  the  cost  per  cycle  Cv  associated  with  c,v.  To  do  so, 
we  use  the  system  of  equations  (6.16)  which  here  takes  the  form 


Cv(x)  =  cv(x)  +pCv(x  +  1), 

x  =  0, 1 

(A.4) 

Cv(x)  =  c,j(x)  +pCv(x  +  1)  +  qCv(x  -  1), 

x  =  2, 3, . . . 

(A.5) 

For  v  =  0, 1  or  v  =  2,  we  use  (A.4)-(A.5)  to  get  (7.8)-(7.9)  by  straightforward  calculations. 
The  case  v  >  3  is  more  involved:  We  observe  that  Cv(x)  =  Cv(x  +  1),  x  =  v, . . .  which  is  readily 
derived  from  the  definition  of  Cv  (which  holds  for  v  >  1).  Moreover,  as  the  relation  (A.5)  implies 

p{Cv(x  +  1)  -  Cv(x))  =  q(Cv(x)  -  Cv(x  -  1)),  x-2,...,v-l  (A.6) 

we  conclude  that 

Cv(x  +  1)  =  (Cv(x  +  1)  -  Cv(x))  +  (Cv(x)  -  Cv(x  -  1))  +  . . .  +  (Cv( 2)  -  C7„(l))  +  Cv(  1) 

x  —  1 

=  '£p-i(Cv(2)-Cv(l))  +  Cv(l ),  x  =  2,...,v-l.  (A.7) 

i= o 

Because  c„(0)  =  <^,(1)  =  0,  we  obtain  (7.10a)  from  (7.7)  and  (A.4),  and  combining  this  last 
relationship  with  (A.7),  we  finally  get  (7.10b)  after  some  algebra. 

Proof  of  Theoem  7.1.  First,  we  observe  from  (A.3)  and  the  hypothesis  that 


oo  oo 

X>WI<4  <KY,(l  +  rv)(l-p)pv  <oo  (A8) 

V  =  0  D  =  0 

because  p  <  1  and  rp  <  1,  and  the  quantity  J  given  by  (4.7)  is  therefore  well  defined.  Next,  using 
(7.6),  and  the  fact  p  <  1,  we  see  that 

OO 

E  7(®)r(s)  -  1  +  ^  P_  p)2  <  oo.  (A. 8) 

Finally,  we  claim  that 

OO  OO 

E'rt*)  E  \c(v)\Cv{x)  <  oo.  (A. 10) 

x=0  v=0 
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Before  giving  a  proof,  we  combine  (A.10)  with  (4.7)  and  (A.9)  to  conclude  that  for  each  x  in  S, 
the  quantity  h(x )  given  by 


oo  oo  oo 

h(x)  :=y 'c(v)hv(x)  =  ^c(u)[C„(a:)  -  JvT(x)\  =  ^c(v)Cv(x)  -  JT(x )  (All) 

■u=0  v=0  v=0 

is  well  defined  since  all  infinite  series  are  absolutely  convergent. 

To  establish  (A.  10),  we  first  interchange  the  order  of  summation  (by  a  simple  application  of 
Tonelli’s  Theorem),  and  note  that 

OO  OO  oo  oo 

53 7(*)  53  \c(v)\Cv(x)  <K  53(1  +  rv)  £y(z)Cv(x) 
x=0  v—0  i7=0  x=0 

oo  oo 

=K(  1  -  P)  53(1  +  rv)  53  fCv(x).  (A  12) 

D=0  x=0 

The  fact  that  the  right  hand  side  of  (A.12)  is  finite  follows  from  (7.8)-(7.10)  once  we  observe  that 
for  v  =  3,4, . . .,  the  bounds 

(pv  x  =  0 

Cv(x)  <  <  Cpv  X(1  -px),  x  =  l,...,v  (A. 13) 

l  C(1  —  pv)  x  =  v,v  +  \,... 

hold  for  some  positive  constant  C  which  depends  only  on  p.  The  calculations  are  tedious  and  are 
omitted;  the  finiteness  of  the  various  infinite  series  follows  from  the  fact  that  p  <  1  and  rp  <  1. 

Combining  (A.9)  and  (A. 10)  with  (A. 11)  we  see  that  h  defined  by  (A.ll)  belongs  to  £>7  =  U^. 
As  the  Poisson  equation  (3.1)  involves  here  only  a  finite  sum,  it  is  immediate  by  substitution  that 
under  the  stated  conditions,  the  pair  (h,  J)  defined  above  is  indeed  a  solution  to  (3.1)  since  for  each 
v  in  5,  the  pair  (hv,  Jv)  is  a  solution  to  the  Poisson  equation.  ■ 

APPENDIX  B:  PROOFS  OF  LEMMAS  9.2-9.3 

Proof  of  Lemma  9.2.  Both  parts  are  proved  along  similar  induction  arguments.  For  the  first 
part,  since  P 9X[T  =  1]  =  pxz{0)  for  all  x  in  S,  the  assumption  (ii)  implies  the  continuity  of  the 
mapping  9  — )■  P^[T  =  k]  for  k  =  1.  The  induction  argument  will  propagate  through  the  relations 

P i[T  =  *  +  !]=  53v/ P*»(*)PJ[T  =  k],  X  E  S  k  =  1,2,...  (B.1) 
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which  are  simple  consequences  of  the  Markov  property  for  all  9  in  0.  Indeed,  fix  x  in  S  and  assume 
that  for  some  A;  =  1,2,...,  the  mapping  9  — >  P y[T  —  k ]  is  continuous  for  all  y  in  S.  By  the  tightness 
condition  (iii),  for  every  e  >  0,  there  exists  a  finite  subset  Fe  of  5\{z}  such  that 

suPee©  [Y,ygFt  P*v(8)\  <£-  ( B .2) 

It  is  now  plain  from  (B.1)-(B.2)  that  for  all  9  and  9'  in  0, 

P i[T  =  k  +  1]  -  P 9X'[T  =  k  +  1]|  <  2e  +  Y,veF'  \p*v(9)Vdy[T  =  k]  -  Pxy{9')Pdy  [T  =  k] |  (B. 3) 

and  therefore  lim^-,#  P X[T  =  k  +  1]  —  P®  [T  =  k  +  1]  <  2e  by  invoking  (ii),  the  induction  hy¬ 
pothesis  and  the  finiteness  of  Fe.  The  continuity  of  9  — >  P®  [T  —  k  +  1]  follows. 

For  the  second  part  of  the  lemma,  only  the  case  y  ^  z  needs  to  be  considered  for  otherwise  the 
result  is  trivially  true  since  Pf[Xt  =  z,T  =  k]  =  0  on  0  for  all  x  in  S  whenever  1  <  t  <  k.  Thus, 
for  all  9  in  0,  set 

q6{x-,t,y)  :=P8x[Xs  ^  z,  l<s  <t,  Xt  =  y],  x,y  e  S  t  =  1,2, ...  {BA) 

and  observe  from  the  Markov  property  that  for  all  x  and  y^z  in  S, 

Pex[Xt  =  y,T  =  k\=Pey[T  =  k-t]-qe{x-,t,y),  1  <t  <  k.  {B.  5) 

Therefore,  by  the  first  part  of  the  proof,  it  suffices  to  show  that  the  mappings  9  -©•  q9{x\t,y), 
t  =  1,2,...,  are  all  continuous  as  x  and  y  range  over  S.  This  is  done  again  by  induction.  Fix  y 
in  S  and  observe  that  for  all  x  in  S ,  the  continuity  of  9  qe{x;t,y )  readily  follows  from  (ii)  for 
t  =  1,  since  qd{x\  1, y)  =  Pex[Xi  =  y\  =  pxy{9).  Now  assume  that  for  some  t  =  1,2,...  the  mappings 
9  q9{v,t,y)  are  continuous  for  all  v  in  S.  The  relations  qe(x;t  +  1  ,y)  =  Y)v^zPxv(6)  •  qe(v,t,y ) 

hold  for  x  in  S  and  t  —  1,2,...;  they  are  simple  consequences  of  the  Markov  property,  and  can  be 
used  to  propagate  the  induction  as  in  the  first  part  of  the  proof.  Details  are  left  to  the  interested 
reader.  ■ 

Proof  of  Lemma  9.3.  The  proof  is  again  by  induction.  Let  x  be  a  given  element  of  S,  and  note 
that  for  t  =  1  the  result  is  true  by  assumption  since  the  tightness  of  {(Xi,P^),  9  e  0}  is  nothing 
but  (iii).  To  proceed  further,  observe  from  the  Markov  property  that  for  every  finite  set  F  of  S, 

P0x[Xt+1<?F]  =  Y,yPxy(O)PeylXt?F},  xeS.  t=l,2,...{B.6) 
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Now  assume  that  for  some  t  —  1,2,...,  the  rvs  {(Xt,Py),  0  6  0}  are  tight  for  each  y  in  S, 
in  which  case  for  every  S  >  0,  there  exists  a  finite  subset  Gy(S)  of  S  such  that  sup@e0  Py[Xt  0 
Gj,(5)]  <  S,  for  y  6  S.  As  in  the  proof  of  Lemma  9.2,  let  Fe  be  the  finite  subset  of  S  guaranteed  by 
(iii)  such  that  (B.2)  holds,  and  set  G(e,8)  =  UyeFaGy(S).  It  is  plain  that 

sup,60  E  yeFp*vW)Kixt  ? G ^ *)]  < 5 


while  the  defining  property  of  Fe  implies 

supSe0  ^2y^FPxy(0)P6y[Xt  0  G(e,<J)]  <  £.  (B. 8) 

Combining  (B.6)  with  (B.7)-(B.8)  leads  to  sup0€0  P®[At+i  0  G(e,  <5)]  <  e  +  £  and  this  completes 
the  proof  since  e  and  6  are  arbitrary.  ■ 
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